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Modernization of the National Airspace System depends critically on the development of 
advanced technology, including cutting-edge automation, controller decision-support tools and 
integrated on-demand information. The Next Generation Air Transportation System national plan 
envisions air traffic control tower automation that proposes solutions for seven problems: 1) 
departure metering, 2) taxi routing, 3) taxi and runway scheduling, 4) departure runway 
assignments, 5) departure flow management, 6) integrated arrival and departure scheduling and 7) 
runway configuration management. Government, academia and industry are simultaneously 
pursuing the development of these tools. For each tool, the development process typically begins 
by assessing its potential benefits, and then progresses to designing preliminary versions of the 
tool, followed by testing the tool’s strengths and weaknesses using computational modeling, 
human-in-the-loop simulation and/or field tests. We compiled the literature, evaluated the 
methodological rigor of the studies and served as referee for partisan conclusions that were 
sometimes overly optimistic. Here we provide the results of this review. 

The FAA’s Next Generation Air Transportation System (NextGen) proposes to modernize the U.S air 
traffic system by deploying advanced technology with the aims of streamlining equipment, consolidating common 
operational tasks and facilitating human management of traffic operations. The FAA has identified specific 
capabilities, referred to as Operational Improvements (01) that will be gradually phased into operations. The OIs of 
interest here are those classified as Decision-Support Tools (DSTs) and procedures for airport tower personnel. 

They are: 

• Departure Metering 

• Taxi Routing & Scheduling 

• Departure Runway Assignment 

• Runway Scheduling 

• Departure Flow Management 

• Integrated Arrival/Departure, and 

• Runway Configuration Management. 

In 2010, the Government Accountability Office (GAO, 2010) recommended that the FAA “identify clear 
goals for the performance of these capabilities or . . . settle on a set of metrics for measuring their performance 
relative to any goals”. In response, the FAA has identified 5 metrics; capacity, efficiency, predictability, safety and 
environment (NextGen Implementation Plan, 2012). The FAA’s Human Factors Research and Engineering Group 
would like for this set of metrics to include measurements of human performance. NASA’s Human Factors 
Research & Technology Division at Ames Research Center has entered into an Interagency Agreement with the 
FAA group to help define these metrics. We refer to the project as the Human Performance Budget. 

NASA has approached the development of a Human Performance Budget in several ways. First, we asked 
a panel of human factors experts to rate whether the proposed OIs could have a positive, or negative, impact on 23 
human performance metrics (Beard, Parke, Holbrook, and Oyung, in preparation). The expert’s ratings indicated 
that overall the OIs could positively influence team situation awareness and coordination if implemented 
appropriately and that one of the most useful capabilities to offer the controllers would be automated support for 
conformance monitoring. The most at-risk metrics identified were controller retention of trained skills and 
knowledge, new tools requiring continuous monitoring for the occurrence of a specified target or event and 
learning/training to efficiency standards. 

NASA then asked how thoroughly the OIs address tower safety problems. We extracted and analyzed over 
200 Aviation Safety Reporting System (ASRS) reports submitted over a 5-year period (Holbrook, Stasio, 
McDonnell, Puentes, Jobe and Beard, 201 1). We found that the majority of the reports dealt with safety threats to 
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runway operations and concluded that proposed OIs addressing improvement in controller situation awareness 
during runway operations could address these concerns. There are no OIs addressing either the oganizational 
climate or inadequate supervision issues identified in the ASRS reports. 

NASA also asked the user community what they need. In a survey involving over 100 tower controllers, 
we asked to what extent the NextGen capabilities could help them in their job or to improve capacity, efficiency, 
flexibility, predictability and safety at their airport. Controllers indicated that departure metering from the ramp 
could be the most helpful tool but that enhanced information would be the most helpful enabler (Holbrook, Parke, 
Oyung, Collins, Gonter and Beard, 2013). 

All three approaches to the Human Performance Budget point to the importance of keeping the controller in 
the loop and providing him with information to help him do his job. This begs the question of whether the advanced 
automation being developed will 

1) support controller and team situation awareness and coordination to aid in decision-making and 

2) be mature enough for implementation by 2018. 

Methods 

Research articles that evaluated tower controllers’ performance using NextGen systems were gathered and 
those articles judged to be of the greatest value were reviewed in considerable depth based on experimental design 
and relevance to NextGen. The goal was to evaluate whether the research and data analyses were experimentally 
rigorous and the conclusions valid. 

Research studies were classified based on their relevance to the seven capabilities listed in the Introduction 
(identified from NextGen OIs) and key enabling technologies required to realize the full potential of those 
capabilities. This report provides a brief description of the meta-analysis of Human-in-the-Loop (HITL) 
publications classified within the seven capabilities of NextGen. Research related to the key enabling technologies 
is evaluated elsewhere (Beard, Galeon and Parke, in preparation). 

Results 

Capability 1: Departure Metering at the Ramp 

The need to reduce airport surface delays has been approached from different angles. Departure metering, 
prior to release from the gate or from the spot, is one approach that has considerable promise and has been 
implemented at several airports. The basic concept of departure metering is very simple: aircraft destined for the 
same runway are provided either gate release times or taxi times from the spot. This provides a way to sequence 
departing aircraft without the many drawbacks of a long physical queue. This idea has been computationally shown 
to be advantageous both operationally and environmentally (e.g., Brinton and Lent, 2012; Brinton, Provan, Lent, 
Prevost and Passmore, 2011; Simaiakis, Sandberg, Balakrishnan and Hansman, 2012). 

The Surface Management System (SMS) is a joint FAA/NASA developed system. An early field test 
focused on OEP judgement (Atkins, Brinton, Walton, Arkind, Moertl and Carniol, 2003). Six years later, the tool 
was still found to provide unreliable pushback predictions (Monroe, 2009) because it did not, at the time, make use 
of airport surveillance data. 

At NASA Ames, an airport surface testbed based on SMS is being used to test the efficacy of tools for taxi 
and runway scheduling (Jung, Hoang, Montoya, Gupta, Malik, Tobias and Wang, 2010). With a predetermined 
sequence and knowledge of the taxi time of a given flight, pushback times can be given to every flight so that it can 
travel unimpeded to the runway and immediately take off. Even though this concept has been tested in high fidelity 
simulation twice, it is still an immature concept. This concept does not account for system uncertainties, such as 
taxi times. In both HITL simulations, the controller had to follow the “advise” of the automation. The initial and 
second HITLs involved two and six controller participants, respectively. Both simulations manipulated traffic level, 
but did not include off-nominal events. Hoang, Jung, Holbrook and Malik (201 1) found that the spot release-time 
recommendations decreased controller situation awareness. Verbal reports from the controllers indicated that 
attending to the automation interfered with their own planning. This happened because of a mismatch between the 
advisor and controller goals. The effects on situation awareness are still be analyzed in a more recent version of the 
advisor that includes electronic flight data (Hyashi, 2012). 

In Europe there is an effort focusing on decreasing gate turnaround time called Airport Collaborative 
Decision Making (CDM). This is a large-scale effort involving airlines, airports, air traffic control and pilots. Each 
aircraft has a predetermined sequence and knowledge of the taxi time of a given flight. Pushback times can be given 
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to every flight so that it can travel unimpeded to the runway and immediately take off. Success of the effort would 
be contingent on adoption of the automation by a large number of airports, and on all parties supplying accurate 
data. 

In the U.S. there is also a research effort focused on the enhancement of continuous, real-time collaboration 
between ramp and ATC (Fernandes, Smith, Spencer, Wiley and Johnson, 2011). Note that this collaboration adds a 
task to the TMC’s repertoire. The Collaborative Airport Traffic System (CATS) is the current test-bed used to 
examine this collaboration. This concept is successfully being used in operations at JFK airport. 

Capability 2: Taxi Routing and Scheduling 

There are several laboratories in the U.S. and Europe testing performance with systems that provide 
automatic generation of taxi routes (Cheng and Foyle, 2002; Stelzer;, Morgan, McGarry, Klein and Kerns, 2011; 
Simaiakis, Khadilkar, Balakrishnan, Reynolds, Hansman, Reilly and Urlass, 201 1). The approach taken in each 
laboratory differs, but in each case two sources of information are required: route data from all aircraft in the 
system, and the results of real-time surface surveillance. 

Testing of recent versions of the Ground-operation Situation Awareness and Flow Efficiency tool (Go- 
SAFE) system (Verma, Kozon, Lozito, Martin, Ballinger, and Cheng, 2010) continue to elicit both functional and 
interface problems (i.e., sub-optimal screen size and resolution, data-tag clutter issues, inappropriate color usage). 
Automated generation and delivery of clearances produced no significant change in workload, in part because of a 
mismatch between the automation-generated clearance plans and controller-generated clearance plans. In general, 
the ad hoc nature of automated taxi route assignments prevented the controllers from developing consistent 
expectations. This is an example of a much more general clash between ad hoc flexibility, typically championed by 
operations research algorithms, and simplicity and consistency, typically championed by human controllers. 

The Go-SAFE software did not allow a sufficiently flexible partnership between the controller and 
automation. As events unfolded, the automated agent would often issue revised routes to the aircraft without 
controller agreement, and notification that the route revision had been issued was not sufficiently clear. Also the 
software did not permit the controller to modify clearances already issued. Finally, there was no way to issue 
conditional clearances. 

Mitre CAASD also has an airport surface testbed (Klein, Stelzer, Nelson, Brinton and Lent, 2010). F1ITL 
simulation results showed that controllers trained to have the same goal as the software always chose to follow the 
taxi routes suggested by software. Controllers stated that their reluctance to issue modified routes was caused by an 
unfriendly keyboard interface. However, when, in a follow-up experiment, the keyboard was augmented by a map- 
based display, controllers remained reluctant to amend automated taxi routes. 

Capability 3: Departure Runway Assignment 

No published HITL simulations were found for the Departure Runway Assignment capability, although 
preliminary discussions of the concept have been reported (Morgan, 2010). 

Subject Matter Expert (SME) knowledge elicitation performed by NASA revealed that traffic management 
coordinators may consider up to 26 pieces of information to make a departure runway assignment. Because it is 
difficult for any human to effectively consider all these factors, particularly under periods of high workload, there is 
a clear need for automation designers to provide decision support tools. The OIs describe tools that suggest optimal 
runway assignments to the controller. He or she would then accept or modify the assignment. However, 
consideringt the length of the list of factors considered by controllers, it is evident that automation will consider only 
a subset of these factors, perhaps a small subset. There may be a temptation for automation to arbitrarily select what 
factors to include, perhaps out a misplaced emphasis on operations-research optimization goals. It is important to 
consider the goals actually weighed by controllers, many of which importantly affect safety (either directly or 
indirectly through a preference for simplicity, which, in things human, tends to promote safety). Where possible 
automation should attempt to collect and integrate information that controllers would assemble manually. It is 
important that there be a clear delineation between factors that that automation has and has not factored into its 
recommendations. This distinction needs to be an important part of training controllers to use such tools. 

Runway balancing is a function performed by the TMC. Atkins and Walton (2002) studied how well 
departure runways were currently balanced. They report that controllers do not currently have accurate information 
about the future departure demand, nor the ability to predict how the surface situation will evolve, both of which are 
needed for effective controller decision-making. 
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Capability 4: Runway Scheduling 

Jung et al. (2010) and Hoang et al. (2011) investigated an automated Runway Scheduler (RS) used in 
conjunction with the SRP mentioned earlier. Together they can produce impressive operational improvements, 
reducing the number and duration of stops in the queue. Surprisingly, however, the expected improvements in 
human performance (workload and situation awareness) did not materialize. As was the case for the SRP, the 
runway crossing strategy of the RS was not always aligned to that of the controller. The unpredictability of ready to 
push-back times, traffic flow constraints and other factors limit the effectiveness of runway scheduling. 

Capability 5: Departure Flow Management 

The goal of the Departure Flow Management (DFM) tool is to use automation to improve the present 
manual process for releasing takeoffs, which requires a tower controller to make a phone call to ARTCC (Spencer, 
Carniol, Pepper and Smith, 2009). In the field trial, the tool was used in shadow-mode. Controllers reported 
benefitting from two features of the tool: a record of what was done, and the seamless integration of new actions into 
the current operational picture. Several interface issues were identified: font size was too small to read the screen 
while standing, and an auditory cue was needed to signal when new information had arrived. A possible advantage 
of the DFM tool is that it provides the means to automatically communicate TFM restrictions to TFDM. This kind of 
automatic sharing of information across tools and across installations is an important potential benefit from 
improved NextGen tools. 

Doble, Timmerman, Carniol, Klopfenstein, Tanino and Sud (2009) present the results of a field trial of a 
DFM tool. Tower controllers judged the tool to be useful, easy to use, and provided good access to needed 
information. Controllers also said that the tool actually opened up more of their time for managing other issues, an 
important “figure of merit’’ that is rarely achieved. 

So far, the Integrated Departure Route Planning (IDRP) tool has only been assessed via Subject Matter 
Expert surveys (Masalonis, Bateman, Song, Taber, Wanke and DeLaura, 2008) therefore no conclusions will be 
drawn here. 

Capability 6: Integrated Arrival/Departure Scheduling 

There were no published HITL simulations found for this capability. 

Arrival/Departure Management Tool (A/DMT) is actually a set of tools that will be a critical part of the 
tower controller’s workstation. A/DMT will integrate information from surveillance (stored in an easily accessible 
database) and other DSTs to characterize arrival/departure demand and surface and airspace constraints. The vision 
is that it will integrate traffic flow constraints provided by DFM with constraints. 

Capability 7: Runway Configuration Management 

There were no published HITL simulations found for this capability. 

The Runway Configuration Management (RCM) problem is to determine which runways should be used 
for arrivals and departures. NASA Langley is developing System Oriented Runway Management (SORM) tools. 
SORM is a composite of two subsystems: Runway Configuration Management (RCM) and the Combined 
Arrival/Departure Runway Scheduling (CADRS) which assigns flights to runways in real time, accomplishing goals 
such as runway balancing (Lohr, Brown, Stough, Atkins, Eisenhawer and Long, 201 1). CADRS software is derived 
from CTAS tools. Fast-time simulation revealed a capacity increase of 0.7% in 2018 and 1% in 2025. However, 
caution is required, as it appears that the capacity gains are achievable only with more frequent dynamic changes 
that may increase complexity and controller workload. 

Discussion 

A great deal of NextGen resources have been channeled toward the difficult job of algorithm development 
and the identification of the NAS benefits (e.g., increased capacity, increased efficiency, reduced environmental 
impact) that can be expected if the algorithms are deployed. The operational and human performance results and 
tool maturity show some successes, such as the departure flow management tool that automates the manual process 
of ATCT phone calls to ARTCC to release take-offs. Unfortunately, there are numerous tools that require 
considerable progress before the NextGen vision can be realized. Four of the seven capabilities have reached a level 
of maturity where they have been tested with humans in the loop; Departure Metering, Taxi Routing and 
Scheduling, Runway Scheduling and Departure Flow Management. 
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For all capabilities current operations have been explored, the proposed concept has been detailed and its 
application discussed, parameters of interest have been identified, and operational benefits that are likely to be 
realized highlighted. Weaknesses in the capabilities development include: 

• Algorithm heuristics were rarely developed from knowledge about the user’s mental model 

• Algorithms often did not plan for uncertainty 

• Initial validation using computational models of the operational system typically did not include 
sufficient (or sometimes any) representation of the human sub-system 

• High fidelity simulations were conducted before tool maturity is adequate 

• HITL simulations neglected to collect both objective and subjective human performance measures 

• HITL simulations disregarded basic experimental design principles, and 

• As the capabilities mature, testing continues in isolation of other capabilities 

It is important that individual tools be reliable. When a single unreliable tool is introduced into tower 
operations, there is a high likelihood that it will not be used as intended. Controllers may use the tool’s 
recommendations during nominal low-workload conditions (where little help was needed anyway), but turn them off 
under off-nominal high workload conditions (precisely those conditions for which it was assumed that help was 
most needed), because the tool’s recommendations for complex situations could not be trusted. Alternatively 
controllers may use a poorly designed tool for an unintended purpose (such as gaining access to raw information 
rather than an action recommendation), a purpose for which some other aid would have been more cost-effective. 
And of course, if using the tool is more trouble than it is worth, controllers may place it under the console and not 
use it at all. 

With the simultaneous introduction of multiple tools anticipated for Nextgen, it becomes even more critical 
to ensure that tools have reached a high readiness level. If unforeseen problems arise, it will be difficult to pinpoint 
which of the multiple new systems introduced is the source of the problem. Furthermore, joint use of multiple new 
tools is likely to produce emergent problems due to unforeseen and unstudied interactions between the tools. Of 
course there is a continuum in the degree to which use of new tools will interact, but in general there is huge overlap 
in the information used by different tools, and in the operational impact of tools on traffic. It is implausible to make 
a “default assumption’’ that multiple new tools developed independently will play well together. The project 
independent nature of the tool development process postpones the issue of properly integrating multiple tools into 
the future, arguably amounting to delaying the resolution of issues— but integration cannot be indefinitely postponed 
if tools are ever to reach operational status. Research into integrated suites of new tools has begun in the last few 
years. A further ramp-up of integration research is needed, including more HITLs testing integrated suites of 
multiple tools, useful for accomplishing multiple task goals. In the long run, integration can become a positive 
strength of tool-development research, providing benefits greater than the sum of individual tool benefits. 
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